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Abstract 

We consider finite irreducible Markov chains. It was shown that mean hitting 
time from one state to another satisfies the triangle inequality. Hence, sum of mean 
hitting time between couple of states in both directions is a metric on the space of 
states. 

1 Paths, sets of paths and probabihties 

Basic definitions and all used properties of Markov chains could be found in [3]. 

Consider Markov chain M with finite state space Z = {zi,...Zn} and matrix of 
transition probabilities P = {pi,j)i,j=i,...,n] P is non-negative and for alH = 1, . . . , n 

n 

Let A*" = (A^, . . . , A™) is distribution of probabilities of states on step m (A° is initial 
distribution). It is known that for all m > 

Let = (Pij'')j,j=i,...,n- Coefficient p^-™'* is the probability that chain with initial 
state Zi will be in the state Zj after m steps. 

Any sequence x = {xq, . . . Xm) of states we call a path of length m. Denote length of 
path and its first and last elements as Len{x) = m, Begin{x) = xq and End{x) = Xm- 

If End{x) = Begin{y), then x ^ y is concatenation of paths x and y. 

Path y is extension of path x, if 

{Len{x) < Len{y)) & (Wi < Len{x)){xi = yi) 

Let S is the set of all paths and, for any m > 0, a,b E Z , 
*e-mail: rozinas@gmail.coni 



1 



S{^,b) = {s e S\{Len{s) = m)k{Begin{s) = a)k{End{s) = b)} 

Asterisk instead of some index in -S'^^b) n^eans dropping of corresponding condition. 
For example, 5"*^ = {s e 5 | Begin{s) = a} . 

For any path x denote by Ext'^{x) set of all extensions of path having length k: 

Ext'^{x) = {s e S\{Len{s) = k)L{s is extension of x)} 

For any set of paths X C 5", let 

Ext''{X) = [j Ext^{x) 
xex 

Sometimes we will write Zj) instead of Pij. For any aEZ,m>0, xE S^^-j define 

m 
i=l 

Evidently, P^{x) is the probability that Markov chain M, being at initial moment in 
state a, will follow path x on first m steps. 
Similarly, for any X C S'^ 

xex 

is the probability that Markov chain M, being at initial moment in state a, will follow 
on first m steps to some path from X. Note that 

„M _ pm(cm \ 
Pa,b — W{a,b)) 

and 

W(m) = EWm)) = 1 (3) 
bez 

For any X Q S define function P on sets of paths X as 

P{X)^Y.P^x)^Y.Pb:U4'^) 

xex xex 

Lemma 1. If X e SJ^^^^ and k > m, then P^{Ext^{X)) = P^{X) 
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Proof. Case k = m is trivial. For k = m + 1 

y&Ext"^+^(X) xeX,z£Z 
m m 

xGX zGZ i=l xeX i=l zeZ 

According to ([T]), 

^p{x„^,z) = 1 
zez 

therefore 

m 

x£X i=l x€X 

For any k > m extension Ext^{X) could be obtained by successive one-step exten- 
sions, therefore lemma hold for all k > m. 

□ 

2 Irreducible finite Markov chains 

Markov chain M is irreducible, if there is positive probability of transition from any state 
to any other (possibly, in more than one step), i.e. 

(W,j)(3m)(pS™)>0) 

For any set of paths X and any W Z denote by X[_vi/] the set of paths from X 
that does not contain states from W, and by the set of paths from X that contain 

at least one element from W: 

X[_w] = {xex\{ywe w){w i x)} 

= {x G X I (3u; G W)[w G X)} 
For one-element sets we will use shortcuts X[-|-^] = X[-|-{^}]. 

Lemma 2. Ij finite Markov chain is irreducible, then for any a,w E Z 

hm Pr(^K.)[-H) = 
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Proof. For any u,v & Z expression P^{S^ *)[+v]) monotonously non-strictly increasing 
function of m or, equivalently, P^{S^ ^'^^_^^) is monotonously non-strictly decreasing. 
Actually, 

According to Lemmas iS';;],)[+.])) = PT^iExt^^iS";^]^^^^^^)), therefore 

{u,*)[+v\) I — (u,*)[+v\) I 

Due to irreducibility, for all m, f G Z exists index v) that satisfies condition 
^ ^ ^ _ g — min {pu^v '^^^ } ■ Theu, for any u,v E Z, 

Hence, for some g > and any u,w & Z 

PtiSl,*)[-^^) < 1 - ? (4) 

Inequality (jl]) means that for any sequence of t steps, probability of event "all states in 
the sequence differ from w" does not exceed 1 — q- Hence, for any sequence, containing 
kt steps, probability of avoiding state w does not exceed (1 — q)'', and for any a,w E Z 

□ 

Corollary 1. If finite Markov chain is irreducible, then exist to, a > 0, 1 > (3 > such 
that for any u,v E Z and any t > to 

P!ii^lu,*)[-v]) ^ 

Proof. Proof is evident from the proof of lemma [2l □ 

Let X is a set of paths and b E W C Z . Denote by the set of paths from X 

that contain elements from W, and b is the first element on the path that belongs to W: 

X[+w:b] = {x e X[+w] I (Vc eW){x e — > x e 

Lemma 3. If finite Markov chain is irreducible, G W Z , and a E Z , then for any 
b eW exist limit 

and 

^Pw,a,b='^ (5) 
b£W 
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Proof, li a G W, statement is trivial and 



1 if 6 = a 
if 6 ^ a 



For any a ^ W, b G W, expression P^iSJ^^-^^^^yy.^) is monotonously non-strictly 
increasing function of m. Really, if mi < 1712, then Ext"^'^{S'^\^^^.^y.f^-^) C S^\-^^_^y^^.j^^ and, 

according to Lemmaffl PriS^^^.)i^^.,,^) = ^a"^^(^^^™^('5(t*)[+H.:.])) < ^a"^^(^(T*)[W:.])- 

Evidently, sets {SJ^^'j[_^_yy.^}bew are pairwise disjoint subsets of S^^^ and hence, ac- 
cording to ([3]), 

J2P^(^Mi+W:,])<^ (6) 

Because addends in ([6]) are non-negative monotonously non-decreasing functions of m, 
limits 

Pw,a.,b = 1™ PnSui,*)[+W:b]) 
m— i>oo V ' /I J 



exist and 



Note that 



therefore 



Qm / I I cm \ I I Qm 

•^(a,*) ~ I U '^{a,*)[+W:b]) ^'^{a,*)[-W] 



b€W 



W(a,*)[+W:b\) — ^ " W{a,*)[-~W\) \' ) 

b<^W 

Select some element c in non-empty set W . From c eW follows C S'^^-)[_^] 

and Pr{Si:^^)[-w]) < PriSj;^.)[-c])- 

According to lemma [21 lim^^oo P™(S'^™^^j_^j) = 0, therefore 

hm Pr(5[:,.)[_w])=0 (8) 

From ([7D and (E]) follows 



and ([5]) proved. □ 

Corollary 2. If finite Markov chain is irreducible, (Z W ^ Z , and a E Z, then for 
any b E W and any m > 1 

Pw,a,b ^ PT{^'{l,*)[+W:b]) 

Proof. Proof is evident from the proof of lemma [31 □ 
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Let W <^ Z, \W\ > 1 , a,b G W and a ^ h. We say that path x is W -arrow from a 
to b, if 

{Begin{x) = a) & {End{x) = b) k, {\fi < Len{x)){xi e W ^ Xi = a) 

i.e. path x leads from a till the first hitting b, and contains no states from W besides a 
and b. Denote by the set of all ly-arrows from a to b. 

Lemma 4. If finite Markov chain is irreducible, W C Z , \W\ > 1, a,b G Z and a^b, 
then 

Proof. By definition, 

Pw\{a},a.b = j™^^a"'('^(t*)[+iy\{a}:6]) 

Note that is the set of paths of length m that start in a, hit some others 

states from iy\{a} , and b is the first one among them. This means that 

^(^,*)l+W\{a}:b] = U Ext"'{x) 

{xeR^\)iz{Len{x)<m) 

All sets of paths Ext'^{x) in the union on the right part are pairwise disjoint, therefore 
according to lemma [H 



Pa{S'^a,.)[-,W\{ay.l>{)=Pa{ IJ ^^^'"(^))= E Pa{Ext^{^))= E 



Leri(x) <m Z/en(x) <m Z/en(a^) <m 



Hence 



lim Pr(5(™.)[+^\W.,]) = lim E ^"(^^x) = E = n< 



b) 



a,o a,o 



Le7i{x) <m 

and proof completed. □ 

According to lemmas |2|31H| P[B^^) is conditional probability of event "6 will be the 
first touched state from iy\{a}, under condition that initial state is a". 



3 Factor chain 

Using lemma [3l introduce, for finite irreducible Markov chains, notion of factor chain. 
Let W Z and \W\ > 1. Consider new Markov chain M with the state space W and 
transition matrix P = (j))a,b<^w, where 

n a = b 
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We will say that M is factor chain of M by set of states W and write M = ISA/W. 
Intuitively, moving from M to M = means 

- Ignore all states that not belong to W, 

- Consider as single step any sequence of steps from some element of W till first 
hitting some other element of W. 

Matrix P is non-negative and, according to lemmas [21 [3l H] and definition (j9]), for any 
a E W satisfies condition 

b£W 

Lemma 5. If finite Markov chain M is irreducible, W Z and \ W\ > 1, then factor- 
chain M = M/iy is irreducible too. 

Proof. Consider any a,b E W, a ^ b. Because M is irreducible, there exists some path 
X = (xo, . . . ,Xm) that leads (in M) from state a to state b and has positive probability, 
i.e. p{xi-i, Xi) > for all i <m. 

Select in the sequence of states x sub-sequence y = {yo, . . . , y^) = {xh(o), ■ ■ ■ , Xh{k)) in 
the following way. Let H{0) = 0. If h{j) already selected and h{j) < m, then select 

hU + 1) = Mtn{t\{t > h{j))k{x, e W)k{xi ^ Xh(j))} 

Evidently, y is subsequence of sequence Xq, . . . , that contains only elements of W , 
each element in y differs from the previous one, y^ = Xh{o) = a and yk = yh{k) = b. 
According to definition of M, corollary [2] and lemma IH for any j < k 

PiVj-UVj) = PiKj-UVj) =Pw\{y^_,},y^_,.y^ ^ _ 1 ( ^^fj,^ _ 1 ,*) _ 1 ] ) 

where t = h{j) — h{j — 1). 
On the other hand, 

^{yj-i,*)[+W\{y,-i}:y,] = 

and 
Hence, 

j<i<h{j) 

and y = {yo, ■ ■ ■ ,yk) is a path in M that leads from a to 6 and has positive transition 
probability from each its state to the next one. Proof completed. □ 
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Note that if Pz^z = for dl\ z G Z and W = Z, then chains M and M coincide. 
From this evident statement follows that transitivity matrix P of factor chain could have 
zero elements outside main diagonal. We could use as a sample M/Z for any irreducible 
Markov chain M, having matrix P with the same property. 

We will use dashed letters for all notations, related to factor chain. For example, S is 
the set of all paths in M. For a = b, according to ([2]), Pa 6 = O5 therefore paths with two 
coinciding adjacent states have zero probability. In following sections we will consider 
only paths from S~^: 

= {xe S\{Len{x) > 0)&(Vz < Len(i))(x,_i ^ Xi)} 

4 Weighted transitions and weighted hitting time 

Hitting time in Markov chain M defined for hitting some set of states (see [3j,p.l2), but 
we will consider hitting time only for sets that contain one state. For any a,b E Z, hitting 
time from a to b is random value Ta,b - minimal index of step when M achieved state b, 
under condition that initial state is a. 

Subject of our interest is the mean hitting time, i.e. function 

00 

H{a, b) = E{Ta,b) = ■ Probability {Tafi = i) 

1=0 

where E denotes mathematical expectation. 

Note that some authors (e.g. [l],p.29) use a little different definition of hitting time 
^: minimal positive index of step when M achieved state b, under condition that initial 
state is a. Evidently, t'^i, — '^a,b for all if a 7^ 6. While Ta,a = for all a, value of ^ 
could be positive. 

For any set of paths X and any a,b E Z denote by X^a,b> the set of paths from X 
that leads from a to b, and contains b only as its end state: 

X<a,b> = {xe X\{x G S(^a,b))Hyi < Len{x)){xi ^ b)} 
Evidently, X^a,b> is the set of all {a, 6}-arrows from a to 6 that belongs to X, i.e. 

X<a,b> = X f] -R^^fe'^^ 

If initial state of Markov chain is a, then the sequence of states till first hitting b will 
follow some random path from S^a,b>, and Ta^b is the length of this path, 

ProbabUityiTa,b = t) = P'aiSl^a,b>) 



8 



and 

H{a,b)^E{Ta,b)^ J2 Len{x) ■ P^^<^\x) 

For proving some properties of function H[a, h) we need to expand considered notion 
and introduce weighted hitting time. 

Select some positive matrix V = ('yij)ij=i,...,n ; (Vi,j = 1, . . . , n)(f jj > 0). Similar 
to matrix P, notations v{zj,Zj) and Vij considered as equivalent. For any path x & S 
define weight of path x with weight matrix V as 

Len{x) 

Weight{V,x) ^ ^ v{xi^i,Xi) 

i=l 

Define Tv,a,b as weight of random sequence of states of Markov chain with initial state 
a till first hitting the state b. Mathematical expectation of rv,a,b is mean weighted hitting 
time from a to b with weight matrix V: 

H{V,a,b)= Weight{V,x)-P!:'<^\x) (11) 

Let £^ is a trivial weight matrix, with all elements equal to 1. Evidently, 

Weight{E, x) — Len{x) 

and 

H{E,a,b) = H{a,b) 

Lemma 6. If finite Markov chain is irreducible, then for any positive weight matrix V 
and states a,b & Z mean weighted hitting time from a to b is finite, i.e. the sum 

H{V,a,b)^ Weight{V,x)-P^'''^^\x) 

is convergent. 

Proof. Order addends of sum according to length of paths: 

oo 

H{V,a,b)= Y Wetght{V,x)P^^<^\x) = J2 E Weight{V,x)P^'''^^\x) 

Let Vmax = '>TT'0'^{vij\i, j = !,•••, n} . Note that for x e S^a,b> holds 

Weight{V, x) < Vmax • Len{x) 
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Evidently, S"^^,^^ C Ext'^{S'^-^^_^-^). Using lemma OQ obtain 
and 

J2 Weight{y,x)P!:'^'~^\x)<v^a.m ^ P'^{x) 



According to corollary [H there are to, a > 0, 1 > f3 > such that for any t > to 

it 

\a,*)[-b]. 



Hence, for any t >to 

J2 Weight{V,x)Pj:''<''\x) < Vmaxm^'^^'^-^^ 

Sum 

oo 
m=to 

is convergent, and this proves the lemma. □ 



5 Direct sum and concatenation on sets of paths 

Concatenation operation ®, considered above as operation on paths, could be considered 
as partial operation on sets of paths. If a G X C S'^*^ and Y C Si^a,*)* , define 
concatenation of sets X and Y as X ^ Y = {x ^ y\{x E X)Sz{y G Y)}. 

Besides, we need another partial operation on sets of paths: direct sum ©, or union 
of disjoint sets. Formal definitions of these partial operations are as follows: 

W = X®Y ^ {XnY = 0)L{Z = XUY) (12) 



W = X^Y ^ (13) 
(3a G Z){{X C ^(*,,,))&(F C S(;,,))&(Z = {x®y\{xe X)L{y G Y)})) 

For any positive weight matrix V and any path x define function 

Len(x) Len(x) 

H^'ix) = Weight{y,x) ■ P^'^^^^ = ( ^ v{x,^,,x,)^ ■ J] p{x,.,,x,) (14) 

1=1 1=1 
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Expand function H^{x) on sets of paths: for any set of paths X C S define 



(15) 



Using definition ([2]) of function P™(x), define function P on any path x 




(16) 



i=l 



and expand function P on sets of paths: 



(17) 



We consider partial function P (partial function H^) defined only on sets X C S", for 
which sum, used in definition, is convergent. 



Lemma 7. If finite Markov chain is irreducible, then for any positive weight matrix V 
partial operations ©, ® and partial functions P{-) {■) satisfy the following properties: 

(a) X®Y = Y®X 

(b) {X ®Y)®W = X ®{Y ®W) 

(c) [x ®Y) ®W = X ®[y ®W) 

(d) X ® {Y ®W) = {X ®Y) ® {X ®W) 

(e) {X ®Y) ®W = {X ®W) ® (Y ®W) 

(f) P{X ®Y) = P{X) + P{Y) 

(g) H''{X®Y) = H^{X) + H^{Y) 

(h) P{X ®Y) = P{X) ■ P(Y) 

(i) H^{X ®Y) = H^{X) ■ P(Y) + P{X) ■ H^{Y) 

As usual, equality of partial functions means that 

- Left and right parts of equahty defined or undefined simultaneously and 

- If left and right parts are both defined, their values coincide. 

Proof. Almost all equalities immediately follow from definition of considered partial op- 
erations and partial functions. Verify statements (h) and (i). Let a G 2', X C SJ^s and 



Note that 



H{y, a,b) = H 




(18) 



Y C SI 



(a,*)- 
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Check (h): 

Lenix) Len{y) 



weX^Y xeX yeY xeX yeY i=l j=l 

Len(x) Len(y) Len(x) 

E( n pi^i-i^^i)Yi n = E ( n M^i-i>2;i)E^(^)) 

xeX i=l yeY j=l xeX i=l yeY 

Len(x) 

^(^) E ( n ^'(^-i' ^^)) = ^(^) E ^ w = ^(^) • ^ w 



i=i xex 
Check (i): 

E ^'^H = EE^''(^®^) = 

weXiSiY xex yeY 

Len(x) Len{y) Len{x) Len(y) 

EE(( E v{xi_i,Xi) + E '"{yj-uVj)) n n p(yj-^^ 

xex yeY i=l j=l i=l j=l 

Len(x) Len(x) Len(y) 



EE(( E '"{^i-i^^i)) n M^i-i'^^i) n 

xex yeY 1=1 1=1 j=i 

Len{y) Lenix) Len(y) 

EE(( E n p{^i-i^^i) n = 

xeXyeY j=i 1=1 j=i 



E( n E (( E ^i^i-i^^i)) n 

ygy j=i xex 1=1 i=i 

Len(x) Len{y) Len(y) 

E( n p(a^i-i'^i)) E (( E n pivj-^^yj)) = 

xex i=l yeY j=l j=l 

Len{y) Lenix) 

E ( n • H'^ix) + E ( n ^^(^^-1'^^)) • H'^iY) = 

yeY j=i xex 1=1 

P{Y) ■ H^{X) + P{X) ■ H^{Y) 

Proof completed. □ 
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Partial operation of direct sum could be expanded on several disjoint sets: 

m m 

r = 0x, ^ (r = |jx,)&(v^,j<m)(z^j^x,ux, = 0) 

i=l i=l 

According to associate property of concatenation (lemma [71 c), concatenation of several 
sets 

m 

Y = ^X,=X^®{X2®{...®X^)...) 

i=l 

does not depends on order of parentheses. Similar to concatenation of two sets, 



r = (^x, ^ (19) 

i=l 

(Vz = 1, . . . ,m - l)(3a, G Z){{X, C 5(:,,^))&(X,+i C Sl^^,^))k 
[Z = {xi ® |(xi eXi)&...&(x„GXj}) 

Lemma 8. If finite Markov chain is irreducible, then for any positive weight matrix V 

and any m > 2 the following properties hold: 

(a) 



(b) 



i=l i=l 

m m 



(c) 



(d) 



i=l 2=1 
m m 

i=l i=l 
mm m 

h''[®x) = Y,[h^{x;).\[p{x, 

i=l i=l J=i 

Proof. Statements (a),(b),(c) immediately follow from lemma [7] ((f), (g), (h) respec- 
tively). Prove (d) by induction by m. For m = 2 statement coincides with lemma [7|^i). 
If (d) true for some value of m, then it is true for m + 1 too, because 



m+l 



^""((8)^0 =^''(((8)^0®^™+0 = H''{®X^)PiXrn+l)+H''{X^+,)P(^(^X, 
i=l i=l i=l j=l 
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m m 

2=1 J = l j = l 



i=l i=i i=i i=l i=i 

Proof completed. □ 

6 Relation between chain and its factor chain 

Consider finite irreducible Markov chain M with set of states Z, some subset W Z, 
\W\ > 1 and factor chain M = 'WL/W. We will use dashed letters for all notations, 
related to factor chain. 

Let Sw is the set of all paths x in M that 

- Have non-zero length, 

- Connect states from W and 

- Last two elements of x that belong to W are different. 

Formally, 

= {x\{x G S)k{Len{x) > 0)L{Begin{x) G 
{End{x) G ^ Endix))} 

Note that Sw is closed under partial operation i.e. 

(ti = X (g) y)k.{x, y G Sw) =^ (v G 5*^^) 

Define mapping ip : Sw — > S of Sw in the set S of all paths in M. 

Let X = {xq, . . . ,Xm) G Sw, 0, = Begin{x) G W and b = End{x) G W. Like in 
the proof of lemma [5l select in the sequence of states x subsequence y = {yo, . . . , yk) = 
ixh(o),-- ■,Xh{k)) as follows: 

Let h{0) = 0. 

If h{j) already selected, h{j) < m and 

(3z)((2 > h{j))k{xi G W)k{xi ^ Xh(,))) 

then 

h{3 + 1) = Min{i\{i > h{j))kixi G W)kixi XhO))} 
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Sequence y satisfies properties: 

- y contains only elements of W; 

- Each element in y differs from the previous one; 

- yo = Xhio) = a; 

- Vk = Xh{k) ^b. 

Define (p{x) = y. Here y considered as path in M. 

Immediately from definition follow that the range of ip is the set S of all paths from 
S, having non-zero length and different adjacent states: 

Range{(p) — — {x & S\{Len{x) > 0)&(Vi < Len{x)){xi^i ^ Xj)} 

It is also evident that >p preserves concatenation: 

{iv, x,y e Sw){v ^x®y^- ip{v) = (p{x) (g) ip{y)) 

Hence, ip is homomorphism of algebraic system < (S^,® > on system < S'^,<S> >■ 
Using homomorphism (p, we could define on Sw equivalence relation 

{x'=^x") {ip{x') = ip{x")) x\x" E Sw 

And equivalence classes 

[Av — ^ Sw\x' —,fX} X e Sw 

Consider again subsequence y = {yo, . . . , yk) = {xh{o), ■ ■ ■ , Xh{k)), constructed in defini- 
tion of (f. According to definition of function h, for alH = 1, . . . ,k segment {xh(i-i) , • • • , Xh(i)) 
of sequence x satisfy properties: 

- Its begin state is Xh(i-i) G W ; 

- Its end state is Xh{i) € W ; 

- Xh{i-i) 7^ Xh(i) ; 

- All elements of segment, besides end one, belong to {Z\W) U {xh(i-i)} ■ 

This means that . . . , Xh(^i)) is VF-arrow from to Xh{i)' 

{xh(i-i),...,Xh(i)) e 

and 

k 
i=l 

Note that (p{{xh{i-i), ■ ■ ■ ,Xh{i))) = ixh{i-i),Xh(i)) and is equivalence class of 

relation : 

^^h(i-i),xh(i) = ii^Hi-i), ■ ■ ■,Xh(i))],f = (p~'^{{xh(i-i),Xh(t))) 
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Here {xh(i-i),Xh(i)) in the rightmost expression considered as a path of length 1 in M. 

Now we will fix weight matrices for chains M and M. For M we will use trivial weight 
matrix E, all elements of E equal to 1. Weight matrix V = (ya,b)a,bew for M define as 
follows: for all couples of states a,b E W in 

- jH^{Kb)/P«b) if (a ^ > 0) 

Va,b = < ■ ^^^^ 

I 1 otherwise 
This means that for all a, 6 G W, a ^ b, if P{B^^ > , then 



Note that ii a,h e W , a ^ h and P(i?^J = 0, then Pi""^'^^(x) = for all x e R^f, 
and hence H^{R^i,) = too. 

Let H^{-) is the function, defined on paths and set of paths in M according to (fT^ 

and ( 1T5|) with usage of trivial weight matrix E, and {■) is the similar function in M, 
using weight matrix V . 

Lemma 9. If finite Markov chain M is irreducible, M = M/Pi^ is factor chain, \W\ > 1, 

and functions ip, H^(-), {■) are as defined above, then 

(a) For all a,b e W ,a ^ b 

lf{{a,b)) = E^{r\{am 

(b) For all a,b,c e W , a b, b c 

H^{{a, b) ® {b, c)) = H'^iv^'iia, b)) ® y,-\{b, c))) 

(c) For any path y = {yo, ...,yk)eS, (Vz < ^ yi) 

_ k 

h'' {y) = H^[^if-\{y.^,,y{))) 
1=1 

Proof, (a) For &ny a,beW,a^b if > 0, then 

If P{Rf,) = 0, then H^'Ha, b)) = Va,b ■ p^. = and H^i^~Hi(^, &))) = H^iKb) = 0- 
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(b) Note that for any a, 6 G W, a ^ h 

Let a,b,c & W, a ^ b, b ^ c. Applying lemma [Tl^i) to concatenation of one-element 
sets of paths (a, b) ® (6, c) in M, receive 

H^'Ha, b) ® (6, c)) = H^'Ha, b)) ■ P((a, b)) + P((a, 6)) • h'' {{b, c)) = 

ff^(^-i((a, b))) ■ p,,, + p,,, ■ H^i^'\ib, c))) = (21) 
H^{v-\{a, b))) ■ P{^-\{b, c))) + P{v-\{a, b))) ■ H^{v'\{b, c))) 

Now (b) follows from lemma [71(i) for sets of paths (f~^{{a,b)) and (f^^{{b,c)) in M. 

(c) Similarly to proof of (b), for any y = (t/q, ...,yk) e S, (Wi < k){yi_i ^ yi) use 
lemma [Hl(d): 

k k ra 

i=l i=l J=i 

km k 

{H^{^'\y^-l, y^)) n nv^\y^~i, %•))) =h\® v-\y^-l, y^) 

i=l J=i i=l 

Proof completed. □ 
Recall that if weight matrix V is given, weighted hitting time from a to 6 is 

H{V,a,b)= ^''(^) 

where S^a,b> = R^^b^ is the set of paths that leads from a to b and contain b only as end 
state. 

Lemma 10. // finite Markov chain M is irreducible, M = M/VT is factor chain, 
\W\ > 1, a,b e W and a^b, then 

H(V, a, b) = H{E, a, b) 

Here HiV , a, b) is weighted mean hitting time in factor chain M, calculated using 
matrix V, defined above (l20D. 
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Proof. In equality 

H{V,a,b)= J2 ^''(^) 

drop from the sum addends for those paths x that have coinciding adjacent states. This 
does not affect the sum, because these paths have zero probabihty. Hence, instead of 
S<a,b> we could use 

^<a,b> ~ ^<a,b> H 5 = (22) 

{x e S\{Len{x) > 0)k{Begin{x) = a)k{End{x) = b)k{\/i < Len(i))(xi_i ^ {xi,b})} 
According to lemma [9]J^c), 

Len(x) 

H{V,a,b)= J2 h''{x)= Y1 H^{<S) (23) 

<a,b> <a,b> 

Note that if x', x" G 5*^^ and x' ^ x" , then sets 

Len{x') Len(x") 

(g) ^~\{xl,,x[)) and (g) ^-\{xl,,x;)) 

i=l 1=1 

are disjoint. Actually, because (p preserves concatenation, any path u that belongs to 
intersection of these two sets must satisfy inconsistent system of equalities: (p{u) = x' 
and (p{u) = x". 

Applying to (|23|) lemma [8]|^b), obtain 

Len{x) 



H{V,a,b) = H'^(^ (g) <^-^((a;,_i,x,)) 



<a,b> 



For completion proof it is sufficient verify equality 

Len{x) 

S<a,b>= <^ V~^{{Xi^l,Xi)) 

■'''^'^ <a,b> 

or 

Leri(x) 



S<a,b> - (g) ^Z-uXi 



<a.b> 
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If X = {xq, . . . , Xk) € S ^> , then all elements in a; = (xq, . . . , Xk) belong to W , a = Xq, 
each element differ from the previous one, and Xk is the only element that coincide with 
b. This means that any path from 



k 

W 



R 



1=1 



leads (in chain M) from a to b, and contains b only at its end. Hence, x G S^a,b>- 

On the other hand, if x = (xq, . . . , Xm) € S^a,b> then, using subsequence y = {yo, . . . ,yk) 
(x/i(o), . . . , Xh{k)) constructed for definition (p, receive 



k 

X = (X) {Xh{i-1), . . -yXhii)) 



i=l 

and 



{xh(i-i),...,Xh(i)) e 



Subsequence y was selected in such a way that {yo, . . . ,yk) G •S'^^^^, therefore 

k Len{x) 
i=l rp"^+ . *=1 

■''^'J <a,b> 

Proof completed. □ 



7 The triangle inequality for mean hitting time 

Theorem 1. In finite irreducible Markov chain, mean hitting time H satisfies triangle 
inequality 

H{a,c) < H{a,b) + H{b,c) (24) 

for any states a, b, c. 

Proof. If at least two of states a, b, c coincide, the statement is trivial. Consider non- 
trivial case of mutually different a, b, c. 

Let W = {a,b,c} . Consider factor chain M = M/Pi^ with weight matrix V, de- 
fined above ([20]). According to lemma [JOl H{a,b) = H{E,a,b) = H(V,a,b), H{b,c) = 
H(y,b,c) and H{a,c) = H(y,a,c). Hence, instead of (124|) we could prove equivalent 
inequality 

H(y, a, c) < H(y, a, b) + Hiy, b, c) (25) 
in very simple factor chain M: it has only three states a, b, c, and p^^^ = Pb,b = Pc,c = 0- 
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Calculate H(y, a, c), H(y, a, h) and H{y , b, c). 



■Len(x) , , 
[X] 



H{V,a,c)= Weight{V,x) ■ 

Here S^a,c> is the set of all paths in W = {a, b, c} that leads from a to c and contains c 
only as its end state. 

Note that 

P(S<a,c>) = 1 (26) 

Formula (126|) asserts that chain M, being in state a, will hit (after one or several steps) 
state c with probability 1. Actually, 

p(s<.,>) = E pr^^V) = hm y: 



_ m— j-oo _ 

Z>en(x) <m 



And, according to lemma [H 



P(5<.,.>) = hm 'K{Ext-{x)) = hm (l - ^ /^(x)) = (27) 

i-iim( 5: p: 



According to lemma [2] 



lim ( V P™(a;) ) = 

^tA(a,*)[-c] 



and proved. 



Evidently, in M any path that leads from a till hitting c, either coincides with (a, c), 
or starts from [a,b). This means that 

^<a,6> = {a, c) © ((a, b) ® :S<b,,>) (28) 

According to ffTSjl . 

F(F,a,6)=:ff''(:S<,,b>) (29) 

therefore from (1251) obtain 



i/(l^, a, c) = i/''(^<a,c>) = i^"" {{a, c) © ((a, 6) © 5<6,e>)) (30) 
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Applying to (15U]) lemma [7|(g,i), obtain 

H(V, a, c) = H^'Ha, c)) + H^'Ha, h) ® 5<,,e>: 



H^{{a, c)) + //""((a, h)) ■ P{S^t,c>) + P{{a, b)) ■ H''{S<k,c>) 



Using (12^ and ([2SD, receive 

(F, a, c) = F^((a, c)) + H^{{a, b)) + P{{a, b)) ■ H(V, b, c) 
Values of function on paths of length 1 directly expressed according (IT^ : 

H^{{a,c)) = v^^^-p^^^ 

H^{{a,b)) = Va,b-Pa,b 

therefore 

H{V, a, C) = Va,c ■ Pa,c + Pa,b ' ^a,b + H{V , 6, c)) 

Swapping roles of a and 6, receive 

H{V, b, c) = Vb,c ■ Pb,c + Pb,a ■ (^6,a + H{V, a, c)) 
Substitute into and get 

H{V, a, C) = Va,c ■ Pa,c + Pa,b ' {va,b + t^6,c " Pb,c + Pb,a " (^fe,a + a, c))) 

and 

(1 - Pa,b ■ Pb,a) ■ H{V, a, C) = Va,c " Pa,c + Pa,b " {Va,b + Vb,c " Pb,c + Pb,a " Vb,a) 

Recall that 

'Pa,b+Pa,c = 1 
Pb,a + Pb,c = 1 
Jc,a+Pc,b = 1 

Note that 

1 - Pa,b ■ Pb,a > 

Actually, otherwise p^^ f, = p^^ ^ = 1; according to fl36|) . ^ = and = 0. 
two equalities contradict to irreducibility of M , asserted in lemma 
Hence, from fl5^ follow 

^(^, a, c) = = = 

Pa,b ' Pb,a 
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Swapping roles of 6, c and a, 6, receive two similar formulas: 

H{V, a, b) = — ^ — — (38) 

Pa,c Pc,a 

^^(^, c) = = = (39) 

Pb,a ' Pa,b 

Substituting these three expressions in f l25|) (and dropping overlines for simpler nota- 
tions) we receive that is equivalent to inequality 

Va,cPa,c + Pa,bVa,b + Pa,b{Vb,cPb,c + Pb,aVb,a) ^ 



1 - Pa,bPb,a 
Va,bPa,b + Pa,cVa,c + Pa,c{Vc,bPc,b + Pc,aVc,a) 

1 Pa,cPc,a 
Vb,cPb,c + Pb,aVb,a + Pb,a{ya,cPa,c + Pa,bVa,b) 

1 - Pb,aPa,b 



(40) 



Because all denominators are positive, for proving (140 p it is sufficient to verify that 
expression g is non- negative: 

g ={l- Pa,cPc,a) {vb,cPb,c + Pb,aVb,a + Pb,a{^a,cPa,c + Pa,bVa,b) 

- Va,cPa,c - Pa,bVa,b " Pa,b{Vb,cPb,c + Pb,aVb,a)) + (41) 
(1 - Pa,bPb,a){Va,bPa,b + Pa,cVa,c + Pa,c{Vc,bPc,b + Pc,aVc,a)) 

Make a series of transformations: 

g =(1 - Pa,cPc,a){Vb,cPb,c + Pb,aVb,a + Pb,aVa,cPa,c + Pb,aPa,bVa,b 
- Va,cPa,c - Pa,bVa,b " Pa,bVb,cPb,c " Pa,bPb,aVb,a) + 
(1 - Pa,bPb,a){Va,bPa,b + Pa,cVa,c + Pa,cVc,bPc,b + Pa,cPc,aVc,a) 



g ={1 - Pa,cPc,a)- 

{Va,b{Pb,aPa,b " Pa,b) + Va,c{Pb,aPa,c - Pa,c) + Vb,a{Pb,a - Pa,bPb,a) + Vb,c{Pb,c " Pa,bPb,c)) + 
(1 - Pa,bPb,a) ■ {Va,bPa,b + Va,cPa,c + Vc,aPa,cPc,a + Vc,bPa,cPc,b) 



g ={1 -Pa,cPc,a)- 

iVa,bPa,b{Pb,a " 1) + Va,cPa,c{Pb,a " 1) + ^^fe,aPfe,a(l " Pa,b) + t^fe,cPfe,c(l " Pafi)) + 
(1 - Pa,bPb,a) ■ {Va,bPa,b + Va,cPa,c + Vc,aPa,cPc,a + Vc,bPa,cPc,b) 
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g =(1 - Pa,cPc,a) ■ {-Va,bPa,bPb,c - Va,cPa,cPb,c + Vh,aPb,aPa,c + Vb,cPb,cPa,c) + 
(1 - Pa,bPb,a) ■ {Va,bPa,b + Va,cPa,c + Vc,aPa,cPc,a + Vc,bPa,cPc,b) 

Consider g as linear form of weights: 

9 =^^a,6(-(l - Pa,cPc,a)Pa,bPb,c + (1 " Pa,bPb,a)Pa,b) + 
^^a,c(-(l - Pa,cPc,a)Pa,cPb,c + (1 " Pa,bPb,a)Pa,c) + 
Vb,a{^ - Pa,cPc,a)Pb,aPa,c + ^^b,c(l " Pa,cPc,a)Pb,cPa,c+ 
Vc,a{^ - Pa,bPb,a)Pa,cPc,a + t^c,fe(l " Pa,bPb,a)Pa,cPc,b 

and do some transformations, using (24) ( 136|) : 

g =Va,b{^ - Pb,c + Pa,cPc,aPb,c " Pa,bPb,a)Pa,b+ 
Va,c{^ - Pb,c + Pa,cPc,aPb,c " Pa,bPb,a)Pa,c+ 
Vb,aO- - Pa,cPc,a)Pb,aPa,c + Vb,c{l - Pa,cPc,a)Pb,cPa,c+ 
fc,a(l - Pa,bPb,a)Pa,cPc,a + Vc,b{l " Pa,bPb,a)Pa,cPc,b 

9 =Va,b{Pb,a + Pa,cPc,aPb,c - Pa,bPb,a)Pa,b+ 
Va,c{Pb,a + Pa,cPc,aPb,c " Pa,bPb,a)Pa,c+ 
Vb,a{^ - Pa,cPc,a)Pb,aPa,c + Vb,c{l " Pa,cPc,a)Pb,cPa,c+ 
fc,a(l - Pa,bPb,a)p a,cPc,a 

+ Vc,b{l -Pa,bPb,a)p a,cPc,b 

9 =Va,b{Pb,ai^ - Pa,b) + Pa,cPc,aPb,c)Pa,b+ 
VaAPbA^ -Pa,b) + Pa,cPc,aPb,c)Pa,c+ 
Vb,a{'^ - Pa,cPc,a)Pb,aPa,c + Vb,c{l - Pa,cPc,a)Pb,cPa,c+ 
Vc,a{^ - Pa,bPb,a)Pa,cPc,a + Vc,b{^ " Pa,bPb,a)Pa,cPc,b 

9 =Va,b{Pb,aPa,c + Pa,cPc,aPb,c)Pa,b+ 
Va,c{Pb ,aPa,c ~l~ Pa,cPc,aPb,c)Pa,c~^ 

t'fe,a(l - Pa,cPc,a)Pb,aPa,c + Vb,c{l - Pa,cPc,a)Pb,cPa,c+ 
fc,a(l - Pa,bPb,a)p a,cPc,a 

+ fc,fe(l -Pa,bPb,a)p a,cPc,b 

Now g is linear form on non-negative weights with non-negative coefficients for each 
weight. Hence, g >0 and proof completed. 

□ 



23 



8 Metric on space of states 

Theorem 2. //M is finite irreducible Markov chain with state space Z, then function 

p{a,b) = H{a,b) + H{b,a) 

is a metric on Z . 

Proof. The theorem asserts that for aU a, 6, c G 2' the following properties hold: 

(a) p(a, 6) > 

(b) p(a, b) = Q ^ a = b 

(c) p{a,b) = p{b,a) 

(d) p{a,c) < p{a,b) + p{b,c) 

Properties (a)-(c) are evident from definition of p. Property (d) follows from the 
theorem [D 

p(a, c) = H{a, c) + H{c, a) < {H{a, b) + H{b, c)) + (i/(c, b) + H(b, a)) = 
{H{a, b) + H{b, a)) + {H{b, c) + H{c, b)) = p{a, b) + p(6, c) 

□ 

Note that p(a, b) has alternative equivalent definition as mean length of minimal loops 
that include both a and b. Set of all such loops is {S<^a,b> ® S<^b,a>) U {S<:b,a> ® S<^a,b>) ■ 

Let Z = {zi, . . . , Zn} is the state space of irreducible Markov chain M with transition 
matrix P = (pj,j)jj=i, Matrix of mean hitting times h = (/ij,j)jj=i,...,n (and hence 
matrix of distances p = (pjj)ij=i,...,n) could be calculated, using theorem for calculation 
mean hitting time of arbitrary subset A ^ Z (p], Theorem 1.3.5]. 

This theorem asserts that vector (/c/^)i=i,...,n of mean hitting time (form state Zi to 
some element of A) is the minimal non-negative solution of the system of linear equations 

kf = ifziEA 

Here "minimal non-negative solution" means that (Vi = 1, . . . , 'ri)(fcj^ < ti) for any 
other non-negative solution (ti)j=i,...,„. 

For any fixed value of index j, apply this theorem to one-element set A = {zj}. and 
receive that vector (/iij)i=i,...,n is the minimal non-negative solution of the system of 
linear equations 

hjj = 

Kj = 1 + Efe^i Pi,khi for i ^ j 
Simplified procedures for calculation mean hitting times proposed by Hunter [2]. 
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